STAT 456: Introduction to Statistical Theory

Lecture 2: Sampling Distributions — \(\bar{X}\), \(S^2\), \(\chi^2\), \(t\), \(F\)

Spring 2026

2026-01-15

Sampling from the Normal Distribution

Today’s Topics:

  • The derived distributions: \(\chi^2\), \(t\), \(F\)
  • Properties of sample mean \(\bar{X}\) and sample variance \(S^2\)
  • The surprising independence of \(\bar{X}\) and \(S^2\) under normality

Setup — Random Sample Notation

Definition (Random Sample)

\(X_1, \ldots, X_n\) are a random sample from population with pdf \(f\) if they are iid with common pdf \(f\).

Sample statistics:

  • Sample mean: \(\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i\)

  • Sample variance: \(S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X})^2\)

Properties of \(\bar{X}\) and \(S^2\) — General Case

Theorem (No normality assumed)

Let \(X_1, \ldots, X_n\) be a random sample from a population with mean \(\mu\) and variance \(\sigma^2 < \infty\). Then:

  • (a) \(E[\bar{X}] = \mu\)
  • (b) \(\text{Var}(\bar{X}) = \sigma^2/n\)
  • (c) \(E[S^2] = \sigma^2\)

Note: Part (c) explains the \(n-1\) denominator in \(S^2\) — it makes \(S^2\) an unbiased estimator of \(\sigma^2\).

Chi-Squared Distribution — Recall

Definition

The chi-squared distribution with \(p\) degrees of freedom, denoted \(\chi^2_p\), has pdf: \[f(x) = \frac{1}{\Gamma(p/2) 2^{p/2}} x^{(p/2)-1} e^{-x/2}, \quad x > 0\]

This is Gamma\((p/2, 2)\).

Lemma (Key Facts about Chi-Squared)

  • (a) If \(Z \sim N(0,1)\), then \(Z^2 \sim \chi^2_1\)
  • (b) If \(X_1, \ldots, X_k\) independent with \(X_i \sim \chi^2_{p_i}\), then \(\sum X_i \sim \chi^2_{\sum p_i}\)

MAIN THEOREM — Properties under Normality

Theorem (Independence and Distributions under Normality)

Let \(X_1, \ldots, X_n\) be a random sample from \(N(\mu, \sigma^2)\). Then:

  • (a) \(\bar{X}\) and \(S^2\) are independent random variables
  • (b) \(\bar{X} \sim N(\mu, \sigma^2/n)\)
  • (c) \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\)

Key observations:

  • Part (a) is surprising! The sum and spread of a normal sample are independent.
  • Part (b) follows from properties of normal (sum of normals is normal)
  • Part (c) requires proof — we’ll do this in detail

PROOF of Independence (Part a)

Goal: Show \(\bar{X}\) and \(S^2\) are independent.

Strategy: Show \(\bar{X}\) and \((X_2 - \bar{X}, \ldots, X_n - \bar{X})\) are independent, then note \(S^2\) is a function of the latter.

Step 1: Write \(S^2\) in terms of deviations

\[S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X})^2 = \frac{1}{n-1}\sum_{i=2}^n (X_i - \bar{X})^2 + \frac{1}{n-1}(X_1 - \bar{X})^2\]

Since \(\sum_{i=1}^n (X_i - \bar{X}) = 0\), we have: \[X_1 - \bar{X} = -\sum_{i=2}^n (X_i - \bar{X})\]

So \(S^2\) is a function of \((X_2 - \bar{X}, \ldots, X_n - \bar{X})\) only.

Step 2: Transform to show independence

WLOG assume \(\mu = 0\), \(\sigma = 1\). Joint pdf of \((X_1, \ldots, X_n)\): \[f(x_1, \ldots, x_n) = \frac{1}{(2\pi)^{n/2}} \exp\left(-\frac{1}{2}\sum_{i=1}^n x_i^2\right)\]

Step 3: Make the transformation

\[y_1 = \bar{x}, \quad y_i = x_i - \bar{x} \text{ for } i = 2, \ldots, n\]

This is linear with Jacobian \(= 1/n\).

Inverse transformation: \[x_1 = y_1 - \sum_{i=2}^n y_i, \quad x_i = y_i + y_1 \text{ for } i \geq 2\]

Step 4: Compute joint density of \((Y_1, \ldots, Y_n)\)

After substitution: \[\sum_{i=1}^n x_i^2 = \left(y_1 - \sum_{i=2}^n y_i\right)^2 + \sum_{i=2}^n (y_i + y_1)^2\]

Expanding and simplifying: \[= ny_1^2 + \sum_{i=2}^n y_i^2 + \left(\sum_{i=2}^n y_i\right)^2\]

Step 5: Observe factorization

\[f(y_1, \ldots, y_n) = \frac{n}{(2\pi)^{n/2}} \exp\left(-\frac{ny_1^2}{2}\right) \exp\left(-\frac{1}{2}\left[\sum_{i=2}^n y_i^2 + \left(\sum_{i=2}^n y_i\right)^2\right]\right)\]

This factors as:

\[= \underbrace{\sqrt{\frac{n}{2\pi}} e^{-ny_1^2/2}}_{\text{density of } Y_1 = \bar{X}} \times \underbrace{g(y_2, \ldots, y_n)}_{\text{joint density of } Y_2, \ldots, Y_n}\]

Since joint pdf factors: \(Y_1 = \bar{X}\) is independent of \((Y_2, \ldots, Y_n)\).

Therefore \(\bar{X}\) is independent of \(S^2\). \(\blacksquare\)

PROOF of Chi-Squared Distribution (Part c)

Goal: Show \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\).

Strategy: Induction on \(n\).

Notation: Let \(\bar{X}_k\) and \(S_k^2\) denote sample mean and variance based on first \(k\) observations.

Key Identity: \[(n-1)S_n^2 = (n-2)S_{n-1}^2 + \frac{n-1}{n}(X_n - \bar{X}_{n-1})^2\]

Base case (\(n=2\)):

\[S_2^2 = \frac{1}{1}\left[(X_1 - \bar{X}_2)^2 + (X_2 - \bar{X}_2)^2\right] = \frac{(X_1 - X_2)^2}{2}\]

So \((n-1)S_2^2/\sigma^2 = (X_1 - X_2)^2/(2\sigma^2)\).

Since \(X_1 - X_2 \sim N(0, 2\sigma^2)\), we have: \[(X_1 - X_2)/(\sigma\sqrt{2}) \sim N(0,1)\]

Therefore: \[S_2^2/\sigma^2 = [(X_1-X_2)/(\sigma\sqrt{2})]^2 \sim \chi^2_1 \quad \checkmark\]

Inductive step:

Assume \((n-2)S_{n-1}^2/\sigma^2 \sim \chi^2_{n-2}\).

From the key identity: \[\frac{(n-1)S_n^2}{\sigma^2} = \frac{(n-2)S_{n-1}^2}{\sigma^2} + \frac{(X_n - \bar{X}_{n-1})^2}{\sigma^2 \cdot n/(n-1)}\]

Now \(X_n\) is independent of \(X_1, \ldots, X_{n-1}\), hence independent of \(\bar{X}_{n-1}\) and \(S_{n-1}^2\).

Computing the distribution of the second term:

\[X_n - \bar{X}_{n-1} \sim N\left(0, \sigma^2 + \frac{\sigma^2}{n-1}\right) = N\left(0, \frac{\sigma^2 n}{n-1}\right)\]

So: \[\frac{(X_n - \bar{X}_{n-1})^2}{\sigma^2 n/(n-1)} \sim \chi^2_1\]

By independence and the additive property of chi-squared: \[\frac{(n-1)S_n^2}{\sigma^2} \sim \chi^2_{n-2} + \chi^2_1 = \chi^2_{n-1}\]

\(\blacksquare\)

Alternative Proof: The MGF/Covariance Approach

Why Multiple Proofs?

Different proof techniques illuminate different aspects of a theorem:

  • Jacobian method: Direct, computational — shows how the factorization works
  • Covariance method: Elegant, conceptual — reveals why independence holds
  • MGF method: Connects to moment theory — useful for identifying distributions

The alternative approach exploits a powerful property of the multivariate normal:

Key Principle

For jointly normal random variables: Uncorrelated \(\Longleftrightarrow\) Independent

This is FALSE for general distributions, but TRUE for normals!

Alternative Proof of Part (a): Independence via Covariance

Goal: Show \(\bar{X} \perp S^2\) using the covariance approach.

Key insight: \(S^2\) is a function of the deviations \((X_1 - \bar{X}, \ldots, X_n - \bar{X})\).

If we show \(\bar{X}\) is uncorrelated with each \(X_i - \bar{X}\), and they’re jointly normal, then they’re independent.

The calculation:

\[\text{Cov}(\bar{X}, X_i - \bar{X}) = \text{Cov}(\bar{X}, X_i) - \text{Cov}(\bar{X}, \bar{X})\]

\[= \text{Cov}\left(\frac{1}{n}\sum_{j=1}^n X_j, X_i\right) - \text{Var}(\bar{X})\]

\[= \frac{1}{n}\text{Var}(X_i) - \frac{\sigma^2}{n} = \frac{\sigma^2}{n} - \frac{\sigma^2}{n} = 0\]

Conclusion: Since \(\bar{X}\) and \((X_1 - \bar{X}, \ldots, X_n - \bar{X})\) are:

  1. Jointly normal (linear combinations of normals)
  2. Uncorrelated (shown above)

They are independent. Since \(S^2\) is a function of the deviations, \(\bar{X} \perp S^2\). \(\blacksquare\)

Alternative Proof of Part (c): The Decomposition Identity

Goal: Show \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\) using MGF ideas.

Step 1: Start with what we know

If \(Z_i = (X_i - \mu)/\sigma \sim N(0,1)\) iid, then: \[\sum_{i=1}^n Z_i^2 = \sum_{i=1}^n \left(\frac{X_i - \mu}{\sigma}\right)^2 \sim \chi^2_n\]

Step 2: The key algebraic identity

\[\sum_{i=1}^n (X_i - \mu)^2 = \sum_{i=1}^n (X_i - \bar{X})^2 + n(\bar{X} - \mu)^2\]

Proof of identity: Expand \((X_i - \mu) = (X_i - \bar{X}) + (\bar{X} - \mu)\) and use \(\sum(X_i - \bar{X}) = 0\).

Step 3: Divide by \(\sigma^2\)

\[\underbrace{\sum_{i=1}^n \left(\frac{X_i - \mu}{\sigma}\right)^2}_{\chi^2_n} = \underbrace{\frac{(n-1)S^2}{\sigma^2}}_{?} + \underbrace{\left(\frac{\bar{X} - \mu}{\sigma/\sqrt{n}}\right)^2}_{\chi^2_1}\]

Step 4: Use independence (from Part a) and MGF uniqueness

  • The two terms on the right are independent (since \(\bar{X} \perp S^2\))
  • If \(U + V = W\) where \(U \perp V\), then \(M_W(t) = M_U(t) \cdot M_V(t)\)
  • We have: \(M_{\chi^2_n}(t) = M_{?}(t) \cdot M_{\chi^2_1}(t)\)

Solving: \((1-2t)^{-n/2} = M_{?}(t) \cdot (1-2t)^{-1/2}\)

Therefore: \(M_{?}(t) = (1-2t)^{-(n-1)/2}\)

This is the MGF of \(\chi^2_{n-1}\)! \(\blacksquare\)

Proof of Part (b): MGF Approach

Goal: Show \(\bar{X} \sim N(\mu, \sigma^2/n)\) using MGFs.

Direct calculation:

\[M_{\bar{X}}(t) = E\left[e^{t\bar{X}}\right] = E\left[e^{t \cdot \frac{1}{n}\sum_{i=1}^n X_i}\right]\]

\[= E\left[\prod_{i=1}^n e^{tX_i/n}\right] = \prod_{i=1}^n E\left[e^{tX_i/n}\right] \quad \text{(independence)}\]

\[= \prod_{i=1}^n M_{X_i}(t/n) = \left[M_X(t/n)\right]^n\]

Since \(X_i \sim N(\mu, \sigma^2)\), we have \(M_X(s) = e^{\mu s + \sigma^2 s^2/2}\).

\[M_{\bar{X}}(t) = \left[e^{\mu(t/n) + \sigma^2(t/n)^2/2}\right]^n = e^{\mu t + \sigma^2 t^2/(2n)}\]

This is the MGF of \(N(\mu, \sigma^2/n)\). By MGF uniqueness, \(\bar{X} \sim N(\mu, \sigma^2/n)\). \(\blacksquare\)

Pedagogical Note

This MGF proof works for any distribution with finite MGF — not just normals! It shows that \(E[\bar{X}] = \mu\) and \(\text{Var}(\bar{X}) = \sigma^2/n\) are universal. The normality of \(\bar{X}\) requires either:

  • Normal population (exact), or
  • Large \(n\) (CLT approximation)

Student’s \(t\) Distribution — Definition and Derivation

Motivation: When \(\sigma\) is unknown, replace it with \(S\):

\[\frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \sim N(0,1) \quad \text{but } \sigma \text{ unknown}\]

\[\frac{\bar{X} - \mu}{S/\sqrt{n}} = \frac{(\bar{X} - \mu)/(\sigma/\sqrt{n})}{\sqrt{S^2/\sigma^2}} = \frac{Z}{\sqrt{V/(n-1)}}\]

where \(Z \sim N(0,1)\), \(V = (n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\), and \(Z \perp V\).

DERIVATION of \(t\) Distribution PDF

Goal: Derive the PDF of \(T = Z/\sqrt{V/p}\) where \(Z \sim N(0,1)\), \(V \sim \chi^2_p\), \(Z \perp V\).

Step 1: Joint PDF of \((Z, V)\)

Since \(Z\) and \(V\) are independent: \[f_{Z,V}(z, v) = f_Z(z) \cdot f_V(v)\]

\[= \frac{1}{\sqrt{2\pi}} e^{-z^2/2} \cdot \frac{1}{\Gamma(p/2) 2^{p/2}} v^{(p/2)-1} e^{-v/2}\]

for \(-\infty < z < \infty\) and \(v > 0\).

Step 2: Transformation

Define: \[t = \frac{z}{\sqrt{v/p}}, \quad w = v\]

This gives us: \[z = t\sqrt{w/p}, \quad v = w\]

Step 3: Compute the Jacobian

\[J = \begin{vmatrix} \frac{\partial z}{\partial t} & \frac{\partial z}{\partial w} \\ \frac{\partial v}{\partial t} & \frac{\partial v}{\partial w} \end{vmatrix} = \begin{vmatrix} \sqrt{w/p} & \frac{t}{2\sqrt{pw}} \\ 0 & 1 \end{vmatrix} = \sqrt{\frac{w}{p}}\]

Therefore: \(|J| = \sqrt{w/p}\)

Step 4: Joint PDF of \((T, W)\)

\[f_{T,W}(t, w) = f_{Z,V}(z(t,w), v(t,w)) \cdot |J|\]

\[= \frac{1}{\sqrt{2\pi}} e^{-t^2w/(2p)} \cdot \frac{1}{\Gamma(p/2) 2^{p/2}} w^{(p/2)-1} e^{-w/2} \cdot \sqrt{\frac{w}{p}}\]

\[= \frac{1}{\sqrt{2\pi p} \, \Gamma(p/2) 2^{p/2}} w^{p/2} e^{-w(1 + t^2/p)/2}\]

Step 5: Marginal PDF of \(T\) (integrate out \(W\))

\[f_T(t) = \int_0^\infty f_{T,W}(t, w) \, dw\]

\[= \frac{1}{\sqrt{2\pi p} \, \Gamma(p/2) 2^{p/2}} \int_0^\infty w^{p/2} e^{-w(1 + t^2/p)/2} \, dw\]

Step 6: Recognize the Gamma kernel

The integrand has the form of a \(\text{Gamma}\left(\frac{p+1}{2}, \frac{2}{1 + t^2/p}\right)\) density.

Recall that for Gamma\((\alpha, \beta)\): \[\int_0^\infty \frac{1}{\Gamma(\alpha)\beta^\alpha} x^{\alpha-1} e^{-x/\beta} \, dx = 1\]

So: \[\int_0^\infty w^{p/2} e^{-w(1 + t^2/p)/2} \, dw = \Gamma\left(\frac{p+1}{2}\right) \left(\frac{2}{1 + t^2/p}\right)^{(p+1)/2}\]

Step 7: Substitute back

\[f_T(t) = \frac{1}{\sqrt{2\pi p} \, \Gamma(p/2) 2^{p/2}} \cdot \Gamma\left(\frac{p+1}{2}\right) \left(\frac{2}{1 + t^2/p}\right)^{(p+1)/2}\]

\[= \frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma(p/2) \sqrt{2\pi p} \, 2^{p/2}} \cdot \frac{2^{(p+1)/2}}{(1 + t^2/p)^{(p+1)/2}}\]

\[= \frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma(p/2) \sqrt{\pi p}} \cdot \frac{1}{(1 + t^2/p)^{(p+1)/2}}\]

\[= \frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma(p/2) \sqrt{p\pi}} \left(1 + \frac{t^2}{p}\right)^{-(p+1)/2}\]

for \(-\infty < t < \infty\). \(\blacksquare\)

Definition (Student’s \(t\))

\(T \sim t_p\) if \(T = Z/\sqrt{V/p}\) where \(Z \sim N(0,1)\), \(V \sim \chi^2_p\), \(Z \perp V\).

PDF: \[f_T(t) = \frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma\left(\frac{p}{2}\right)\sqrt{p\pi}} \left(1 + \frac{t^2}{p}\right)^{-(p+1)/2}\]

Note: \(t_1 = \text{Cauchy}\) (no mean!)

Properties of \(t\) Distribution

  • \(E[T_p] = 0\) if \(p > 1\)
  • \(\text{Var}(T_p) = p/(p-2)\) if \(p > 2\)
  • As \(p \to \infty\): \(t_p \to N(0,1)\)
  • Symmetric about 0
  • Heavier tails than normal

Snedecor’s \(F\) Distribution — Definition

Motivation: Compare variances from two populations

If \(X_1, \ldots, X_n \sim N(\mu_X, \sigma_X^2)\) and \(Y_1, \ldots, Y_m \sim N(\mu_Y, \sigma_Y^2)\) independent samples:

\[F = \frac{S_X^2/\sigma_X^2}{S_Y^2/\sigma_Y^2} = \frac{\chi^2_{n-1}/(n-1)}{\chi^2_{m-1}/(m-1)} \sim F_{n-1, m-1}\]

Definition (Snedecor’s \(F\))

\(F \sim F_{p,q}\) if \(F = (U/p)/(V/q)\) where \(U \sim \chi^2_p\), \(V \sim \chi^2_q\), \(U \perp V\).

PDF: \[f_F(x) = \frac{\Gamma\left(\frac{p+q}{2}\right)}{\Gamma\left(\frac{p}{2}\right)\Gamma\left(\frac{q}{2}\right)} \left(\frac{p}{q}\right)^{p/2} \frac{x^{(p/2)-1}}{\left(1 + \frac{p}{q}x\right)^{(p+q)/2}}\]

Relationships Between \(\chi^2\), \(t\), \(F\)

Theorem

  • (a) If \(X \sim F_{p,q}\), then \(1/X \sim F_{q,p}\)
  • (b) If \(T \sim t_q\), then \(T^2 \sim F_{1,q}\)
  • (c) If \(X \sim F_{p,q}\), then \(\frac{(p/q)X}{1 + (p/q)X} \sim \text{Beta}(p/2, q/2)\)

Summary Diagram

\[N(0,1) \xrightarrow{\text{square}} \chi^2_1 \xrightarrow{\text{sum of } p} \chi^2_p\]

Derived distributions:

\[t_p = \frac{Z}{\sqrt{\chi^2_p / p}} \qquad \text{where } Z \perp \chi^2_p\]

\[F_{p,q} = \frac{\chi^2_p / p}{\chi^2_q / q} \qquad \text{where } \chi^2_p \perp \chi^2_q\]

Key results:

  • \(\bar{X} \sim N(\mu, \sigma^2/n)\)
  • \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\)
  • \(\bar{X} \perp S^2\) (under normality)
  • \((\bar{X} - \mu)/(S/\sqrt{n}) \sim t_{n-1}\)